Speedup regr_test.py
by running test cases concurrently
#10714
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As we add more and more test cases,
regr_test.py
is getting kinda slow. Other than mypy_primer, it's now the slowest CI job we have by a long way (and we don't run mypy_primer on all PRs -- for example, it's skipped on this PR!).The reason for the slowness is that we now have regression tests for 11 stubs packages. We run all regression tests on Python 3.8-12 inclusive, and we run them all on linux, darwin and win32. That adds up for a total of 165 subprocesses that are created for each run of the test when it's run with
--all
(the flag we use in CI).At some point we may want to consider sharding this test between GitHub Actions workers, similar to the way we run
mypy_test.py
and pyright in CI. (We can also possibly reconsider whether we need to, e.g., run all tests on darwin, linux and Windows). For now, though, we can speed things up a lot just by running the subprocess concurrently using aProcessPoolExecutor
. This cuts the execution time in CI roughly in half, from around 5-6 minutes to around 2-3 minutes.(N.B.: A
ProcessPoolExecutor
feels like a slightly blunt instrument here with a lot of overhead; I'm sure there are more efficient ways of spawning subprocesses concurrently. I got this to work reasonably quickly, however, and when I tried different approaches I quickly ran into race conditions. I think this is ~good enough for now.)